Fix Gate plugin cache recovery and active project roots - #1

…bases

Surfaced while running the first end-to-end demo on the public Facebook
React monorepo. Every JavaScript file >32 KB threw "Invalid argument" from
the tree-sitter Node binding and fell through to regex fallback. The binding
ships with a ~32 KB string buffer; production source files routinely exceed
that. The project documented "native AST compression for JS/TS" but in
practice almost no real file ever hit the AST path.

Fix: replace parser.parse(source) with the chunk-callback API
(parser.parse(callback)) which streams 4 KB slices and has no cap. Verified
on the React monorepo: 165-file reconciler benchmark went from 165 AST
failures to 0, and the full packages/ tree (2,080 files, 3.93M tokens)
compresses to 306k tokens with all AST signatures intact (92% reduction).

Added src/scripts/benchmark-real-repo.ts so anyone can reproduce on any
directory: token counts, per-language breakdown, top-10 expensive files,
cost estimates against Claude/GPT-4o/GPT-5.

- src/lib/astParser.ts: chunk-callback parsing, 4 KB chunks, doc comment
- src/scripts/benchmark-real-repo.ts: new benchmark harness
- README.md: v0.3.1 note, real-codebase benchmark table
- package.json + main.ts: 0.3.0 -> 0.3.1
- Tests still pass: 13/13 unit, 55/55 stress

After v0.3.1 fixed the tree-sitter buffer limit, a symbol-recall fidelity
test on the public Facebook React monorepo (1,010 files, 7,047 exported
symbols) revealed the compression was still LOSSY in non-obvious ways:

  Pre-fix recall: 68.7%  (439 of 1403 symbols lost on reconciler alone)
  Post-fix recall: 99.1% (60 of 7,047 symbols lost across full React)

Four distinct bugs were fixed:

1. Flow type syntax broke the JS grammar. Files with `@flow` (most of
   Meta's source) used Flow generics like `<+T>` that tree-sitter-javascript
   rejected. Now: detect `@flow` pragma in first 4 KB and route to the
   TSX grammar, which parses Flow with ~0 errors (Flow ≈ TS minus variance
   markers; TSX adds JSX which Flow files frequently use).

2. ERROR-root parse trees emitted junk function nodes. When the grammar
   gave up on a file, error-recovery wrapped `if`, `then`, `switch`
   keywords as fake `function_declaration` nodes. Our extractor recorded
   them as real exports. Now: if `tree.rootNode.type === "ERROR"`, fall
   back to regex extraction immediately.

3. Multi-line `export { A, B, C } from './x'` blocks lost every name. The
   collector took only `node.text.split("\n")[0]` which yielded just
   `export {`. Now: collapse whitespace and keep the full block up to 4 KB.

4. CommonJS export forms were invisible. `module.exports.foo = ...` and
   `exports.foo = ...` parse as plain assignment_expressions, not exports.
   React's npm shim files are 100% CJS. Now: supplement the AST output
   with a CJS pattern scan, and extend the regex fallback to match.

Adversarial review (FAIROS Principle 4):
  Before this change, gatemcp's docs claimed "92-97% input-token reduction"
  but the compressed view was silently dropping ~31% of real exports on
  large production codebases. That is lossy compression masquerading as
  semantic. The honest post-fix numbers on facebook/react are:
    - 80% token reduction (was 92%, but lying)
    - 99.1% symbol-recall fidelity (was ~69%)
  Faithful 80% is far more useful to an LLM than lossy 92%.

New tool: src/scripts/fidelity-test.ts measures symbol recall on any
directory by comparing AST-extracted symbols against a ground-truth regex
on the raw source. Fails CI when overall recall falls below 95%.

- src/lib/astParser.ts: detectFlowFile, pickGrammarLanguage,
  ERROR-root guard, multi-line export capture, CJS augmentation
- src/scripts/fidelity-test.ts: new validation harness
- README.md: honest benchmark table + reproduction commands
- package.json + main.ts + scripts: 0.3.1 -> 0.3.2
- Tests still pass: 13/13 unit, 57/57 stress

After Experiment #4b (Cursor-as-LLM round-trip test on dedupContext.ts),
the compressed view showed something embarrassing: every `export function`
declaration was duplicated FULL-BODY inside the Exports section. The
Functions section already held the same signature. We were spending
~25% of compressed tokens on redundant copies of function bodies.

For dedupContext.ts (282 lines, 2,038 tokens raw) the impact was severe:
  Before: 1,523 tokens compressed (25% reduction)
  After:    251 tokens compressed (88% reduction)

Full facebook/react monorepo numbers move with it:
  Before: 791 k compressed tokens (80%)
  After:  446 k compressed tokens (89%)
  Saving:  +345 k tokens, +$1.04 per Sonnet 4 full-context query

Fidelity stays unchanged at 99.1% — we still record an "export function
foo(args)" marker for every export, just without the body.

Change is localized to collectJsTsNode. When an export_statement wraps a
function/class/interface declaration, capture only the first line
(the export-prefixed signature). When it wraps a re-export block, type
alias, default expression, or lexical declaration, keep the full text
(those carry information that isn't recovered elsewhere).

Also adds src/scripts/cursor-llm-test.ts — a single-file harness that
renders the compressed view of any file plus four validation prompts to
try in a fresh Cursor chat. Surfaced this bug; staying in the repo for
future audits.

- src/lib/astParser.ts: split export_statement handling by wrapped type
- src/scripts/cursor-llm-test.ts: new harness
- README.md: updated benchmark numbers (89% reduction, 99.1% fidelity)
- Tests still pass: 13/13 unit, 59/59 stress

Append "Last reviewed: 2026-05-15" trailing comments to 17 files that
were not touched during the v0.3.0 -> v0.3.2 work but were re-audited
alongside the fidelity test pass. No behavior change.

- 13 TS modules (lib/, tools/, exp2/exp3, scale-test, test, measure-schemas)
- 2 docs (TROUBLESHOOTING.md, competitive-analysis.md)
- .gitignore, tsconfig.json (JSONC comment, parses cleanly via tsc)

Verified: tsc --noEmit clean, tsc --showConfig parses tsconfig.

Update copyright line from generic "Gate-MCP Contributors" to
"Aaron Mecate and Gate-MCP Contributors" to accurately credit the
project author while preserving the contributor language. MIT license
terms and detection pattern unchanged.

The session dedup cache that backs gate_compress_file's ~93% reread
savings was previously an in-memory Map that vanished every time the
MCP server restarted. With multiple IDEs (Cursor, Windsurf, Claude
Code) often running against the same project, this also meant zero
sharing between sessions.

Phase 2 SQLite migration:

  * New src/lib/cacheDb.ts encapsulates better-sqlite3 setup, schema,
    CRUD, LRU eviction, and graceful shutdown. The raw Database object
    never leaks; callers only see typed CacheEntryRow values.
  * DB lives at <projectRoot>/.gate-mcp/cache.db by default (override
    via GATE_CACHE_DB). Path is validated through pathGuard.safeResolve
    so a hostile env var cannot point us at /etc/passwd.
  * WAL journal_mode + NORMAL synchronous makes concurrent IDE access
    safe without sacrificing write throughput.
  * Schema is intentionally minimal:
      cache_entries(file_path PRIMARY KEY, hash, content, tokens,
                    original_tokens, type, hit_count, updated_at)
      + idx_updated(updated_at) for LRU eviction.
  * better-sqlite3 is an OPTIONAL dependency. If the native binary
    fails to load (compile failure, missing prebuild for the platform),
    the cache transparently degrades to an in-memory Map with the
    exact same API and semantics. The MCP server keeps running.
  * LRU eviction: 10,000 entries OR 500 MB of content, whichever hits
    first. Constants live at the top of cacheDb.ts.
  * totalTokensSaved is now derived from
    SUM(hit_count * (original_tokens - tokens)) instead of being
    bumped on every hit -- cleaner and consistent across processes.
  * src/main.ts wires closeCacheDb() into the existing SIGINT/SIGTERM
    graceful-shutdown path next to terminateOcr().
  * src/tools/dedupContext.ts rewritten to delegate to cacheDb. All
    public signatures (checkCache, storeInCache, handleDedupContext
    with check/store/stats/clear actions) are unchanged so the existing
    13 unit + 61 stress tests continue to pass.

Tests:

  * 4 new unit tests (Test 14-17): store-then-check increments
    hit_count, file mutation triggers cache_update, stats consistency,
    clear wipes everything. Tests pass under both SQLite and Map
    backends.
  * 1 new stress scenario: 1,000 stores + 10,000 checks (~80% target
    hit ratio). Measured ~0.1ms/store and ~0.06ms/check on the
    SQLite backend.

Verified, no regressions:

  * Unit:    17/17 passing (was 13/13)
  * Stress:  63/63 passing (was 61/61)
  * Fidelity test on facebook/react packages/ (1,010 files):
      symbol-weighted recall 99.1% (6,986 / 7,047) -- unchanged.
  * Benchmark on facebook/react packages/ (2,080 files, 3.93M tokens):
      89% reduction -> 445.8k tokens -- unchanged.

src/lib/astParser.ts and src/lib/symbolGraph.ts are untouched.

These directories hold internal planning notes, architecture deep-dives,
mentor reports, research logs, and competitive analysis that are kept
locally for development reference but not appropriate for the public
GitHub repository. Files remain on disk; only git tracking is removed.

README
  - Collapsible <details> blocks for per-IDE config (Cursor, Claude Code,
    Windsurf, Antigravity, VS Code Copilot, plus a generic catch-all).
    Each shows the verbatim mcp.json snippet using npx -y gatemcp so
    users get one-line install after npm publish.
  - All historical "Note (v0.x.x)" blocks moved into a Changelog section
    of collapsible <details> at the bottom. Top of README now leads with
    the value prop, not the patch log.
  - Worked-example token math collapsed under a <details>.
  - Source install collapsed under a <details>.
  - Header links and footer now link to the new website
    (gate-mcp-site.vercel.app) alongside Install/Tools/Benchmarks.

package.json (npm publish prep)
  - homepage -> https://gate-mcp-site.vercel.app/
  - repository -> github.com/Dukeabaddon/Gate-MCP
  - bugs -> issues URL
  - author -> Aaron Mecate
  - files -> [dist, README.md, LICENSE] so npm pack only ships runtime
    artifacts (current tarball 97.3 KB, 103 files)
  - prepublishOnly -> clean + build + test (no broken publishes)
  - keywords expanded for npm discovery (llm, ast, tree-sitter, ...)

Verified: 17/17 unit, 63/63 stress. Pack dry-run clean. GitHub repo
About sidebar updated to point at the website too (via gh repo edit).

npm publish succeeded after creating the @gatemcp organization
(the unscoped name "gatemcp" is rejected by npm's similarity check
against Gate.io's pre-existing "gate-mcp" package). The CLI binary is
still named "gatemcp" so terminal usage is unchanged; only the package
name on the registry differs.

Changes
  - package.json: name -> @gatemcp/cli
  - README.md: install command + all 6 IDE config snippets updated to
    use "@gatemcp/cli", changelog entry expanded to explain the scope,
    roadmap "npm publish" checkbox flipped
  - src/test.ts + src/tools/help.ts: stale "v0.2.0-alpha" version
    strings updated to v0.4.0 (caught from the publish-time test output)

Verified
  - Published: + @gatemcp/cli@0.4.0 (97.3 KB tarball, 103 files)
  - npm view @gatemcp/cli returns clean metadata, homepage = website
  - Fresh install: npm install @gatemcp/cli@0.4.0 -> 227 deps in 7s,
    .bin/gatemcp symlinked correctly, shebang intact
  - Unit tests still 17/17 passing, stress 63/63

Introduces transparent compression of every other MCP server the user
has configured. The LLM sees one compressed catalog through gatemcp
instead of paying full schema cost (~3K tokens per server) for each of
them every turn. On a typical 10-server / 50-tool roster this cuts
per-turn MCP schema overhead by 70-90%.

How it works
  - User drops .gate-mcp/proxy-servers.json in their project root
    (same shape as their IDE's MCP config — copy-paste works).
  - gatemcp spawns each downstream MCP server as a child stdio client
    lazily, the first time a tool from that server is referenced.
  - gate_proxy_tools returns a compressed catalog (TOON-tabular,
    schemas rendered as "name:type[]" rather than full JSON Schema).
  - gate_proxy_tools action='describe' returns the full schema for
    one specific tool — the LLM only pays that cost just before
    invoking, not for every tool in the catalog.
  - gate_proxy_call forwards a tool call through the connection pool
    and pipes the response through the existing TOON compressor from
    gate_clean_response.

Safety / FAIROS adversarial review
  - Per-call timeout (default 30s, override via GATE_PROXY_TIMEOUT_MS
    env or timeoutMs arg). Hung downstream servers cannot starve the
    parent process.
  - Wedged connections are dropped synchronously on timeout but cleanup
    of the child process is fire-and-forget so the LLM gets the error
    immediately instead of waiting another 1-3s for the child to die.
  - Concurrent callers requesting the same server share one spawn
    promise (no double-spawn race).
  - Graceful shutdown closes every live downstream connection.
  - Config loader validates JSON shape and surfaces clear errors
    pointing at the config path.

Added
  - src/lib/proxyClient.ts (connection pool, config loader, timeout)
  - src/tools/proxyTools.ts (handleProxyTools + handleProxyCall)
  - src/scripts/mock-mcp-server.ts (test fixture, not shipped to npm)
  - .gate-mcp/proxy-servers.example.json (sample config, committed)
  - 8 new tests in src/test.ts (18-24a) — spawn, list, describe, call,
    TOON compression, status, timeout, missing-server error
  - help.ts entries for both new tools + tool directory bumped to 9

Tarball cleanup
  - package.json files field now uses explicit globs + negations so
    test runners and fixtures are excluded from the npm tarball.
    Tarball shrank from 116.9 kB to 63.7 kB (-46%).

Tested
  - 25/25 unit (was 17/17)
  - 69/69 stress (unchanged)
  - Timeout fires in ~252ms with 250ms limit; cleanup is non-blocking
  - Mock server cold spawn: ~1s, warm calls: 1ms

Version bump 0.4.0 -> 0.5.0. Not yet published to npm (publish needs
2FA OTP which is currently blocked).

Co-authored-by: Cursor <cursoragent@cursor.com>

gate_validate_compression + validate-llm CLI (mock/ollama/openai providers).
Four unit tests; scoring: recall 40%, usage 35%, specificity 25%.

Optional tree-sitter grammars: PHP, Ruby, Kotlin, Bash, Swift (+ Vue/Svelte/YAML
deps documented; regex fallback when native load fails). test-fixtures/tier2/*.

vscode-extension/: MCP JSON snippets for Cursor + generic mcp config.

README: known limitations (graph baseline, Flow heuristic, OCR auto, memory JSON).
Tests: 29 unit, 85 stress.
Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

gate_memory now stores KV in memory_entries inside .gate-mcp/cache.db
(same WAL file as dedup). JSON fallback when better-sqlite3 unavailable.
One-time import from memory.json → memory.json.migrated.

README: strike Leiden, Ollama routing, tool-result cache; mark core scope done.
Known limitations updated. Tests: 30 unit (+ migration), 87 stress.

Verified on /Users/macbookair/demo/react: 86% token reduction (6.48M → 925k).

Co-authored-by: Cursor <cursoragent@cursor.com>

Keep the GitHub repo focused on the shipped product:

Removed from tracking (still local where noted):
  - DEMO_SCRIPT.md — hackathon/video pitch only (gitignored, file kept on disk)
  - src/exp2-semantic.ts, src/exp3-toon.ts — FAIROS one-off experiments
  - src/measure-schemas.ts — schema token measurement script
  - src/scale-test.ts — local scale benchmark harness
  - src/scripts/cursor-llm-test.ts — superseded by validate-llm.ts

Already excluded via .gitignore (unchanged policy):
  - docs/, documentation/, graphify-out/, vendor/, .gate-mcp runtime data

Public repo retains: src product code, test.ts, stress-test.ts,
benchmark-real-repo, fidelity-test, validate-llm, mock-mcp-server (tests),
test-fixtures/tier2, vscode-extension/, proxy-servers.example.json.

Tests: 30/30 unit, 77/77 stress after cleanup.
Co-authored-by: Cursor <cursoragent@cursor.com>

Wire graphify-out/GRAPH_REPORT.md into gate_graph_query with nested path
discovery, graphify_hubs/search/map query types, and symbol-search fallback
when communities or hub names miss the tree-sitter index.

Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

- Honest savings when compression expands output; YAML structure mode
- graphify_map baseline from full GRAPH_REPORT.md; stale graphify warning
- gate_session_stats, gate_init health, optional graphify update on rebuild
- AlgoTrading regression script (npm run validate:algo)

Co-authored-by: Cursor <cursoragent@cursor.com>

Document gate_init, gate_session_stats, structure depth, honest
metrics, graphify_map baseline, recommended_stack workflow, and
validate:algo script.

Co-authored-by: Cursor <cursoragent@cursor.com>